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SPECIFICATION 
AUTOMATED EVALUATION SYSTEM AND PROGRAM 

TECHNICAL FIELD 

5 The present invention relates to an automated evaluation system that 

automatically evaluates an application program that operates on a target system 
such as a micro computer (heroin after roforrcd t o as a microcomputer)^ according 
to input events such as key inputs and reference outputs such as output pictures 
and the like with respect to the input events. 

10 

BACKGROUND TECHNOLOGY 

In recent years, microcomputers a^e — have been w idely used and 
implemented in a variety of apparatuses such as home electric appliances. A 
microcomputer has a built-in ROM (Read Only Memory) in which an application 

15 program is writtcn stored , in order to operate according to the specification of an 
apparatus in which the microcomputer is implemented or peripheral 
apparatuses therefor. Also, liquid crystal display apparatuses (hereinafter 
referred to as LCDs "Liquid Crystal Displays") are implemented in a variety of 
apparatuses such as home electric appliances. Therefore, when the user inputs 

20 an input event such as a key input, a microcomputer outputs to a LCD an output 
picture corresponding to the input event according to an application program. In 
this respect, in the event of confirming the operation of the application program 
by the microcomputer, output results for numerous input events according to the 
specification of an apparatus in which the microcomputer is implemented must 

25 be confirmed. 

Accordingly, when a microcomputer is developed, not only development of 
its hardware is important, but also development of an application program is 
important. For development of an application program, an InCircuit Emulator 
(hereinafter referred to as ICE "InCircuit Emulator") or the like is used. The 

30 ICE can emulate an operation of an application program on a target board. As 
described above, for confirming the operation of an application program, there 
are numerous input events that are to be confirmed. Therefore, when the 
operator directly inputs input events using the ICE, it takes a long time to input 
them, and there is a possibility that operator may make input errors. Also, when 

35 the operator confirms input events and emulated results by picture outputs, such 
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a work takes a long time and there is a possibility that the operator may make 
confirmation errors. 

Furthermore, in order to evaluate the application program by confirming 
its operation, the same events are repeatedly evaluated in order to improve the 
5 evaluation accuracy. Therefore, in order to confirm the operation of the 
application program more efficiently at a higher accuracy, an automated 
evaluation system that can automatically and repeatedly input numerous input 
events and that can automatically evaluate output results corresponding to the 
input events is used. 

10 Fig. 4 shows a structure for performing an automated evaluation of an 

application program of a conventional microcomputer. An automated evaluation 
system 50 is composed on a personal-purpose computer (hereinafter referred to 
as a personal computer) PC, and is capable of accessing a disk apparatus DU of 
the personal computer PC. Furthermore, the automated evaluation system 50 is 

15 equipped with an emulation expansion board 50a in order to enter input events 
in a target board TB. On the other hand, an ICE 51 is connected to the personal 
computer PC through an RS 232 and also to the target board TB. Also, a 
debugger 52 is implemented on the personal computer PC, such that, while the 
ICE 51 is emulating an application program AP, the debugger 52 can perform 

20 debugging. In the event of automatic evaluation of the application program AP, 
first, the operator prepares an input event file IF that stores a plurality of input 
events and a reference output file OF that stores reference output results 
corresponding to the input events. Then, when the automated evaluation system 
50 is started, the automated evaluation system 50 reads an input event stored in 

25 the input event file IF and a reference output result stored in the reference 
output OF from the disk drive DU. Then, the automated evaluation system 50 
enters the input event through the emulation expansion board 50a to the target 
board TB. 

Then the ICE 51 emulates the operation of the application program AP 
30 according to the input event, and outputs an output result to the automated 
evaluation system 50. Then, the automated evaluation system 50 compares the 
output result and the reference output result to determine whether or not the 
output result is correct, and outputs a determination result to a result log file or 
the like. Furthermore, the automated evaluation system 50 automatically 
35 performs the input, comparison and determination steps repeatedly for a 
plurality of input events and automatically evaluates the application program 
AP. 
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However, in the automated evaluation by the automated evaluation 
system 50 and ICE 51, data is transferred between the automated evaluation 
system 50 and the target board TB, and therefore a function for automated 
evaluation needs to be implemented. In other words, the automated evaluation 
5 system 50 needs the emulation expansion board 50a in order to convert input 
events to data for emulation. Furthermore, the target board TB is provided with 
an expansion board interface TBa implemented therein in order to receive input 
events and a personal computer interface TBb for outputting output results to 
the personal computer PC. Also, an automated evaluation system interface 

10 program IP is implemented in the application program AP in order to transfer 
data to a VRAM ( Vioual V ideo Random Access Memory) for analyzing the input 
events from the automated evaluation system 50 or outputting output results to 
the personal computer PC. As a result, the application program AP to be 
evaluated by the automated evaluation is different from an application program 

15 that is actually implemented in the microcomputer, and the automated 
evaluation system interface program IP adds a load to a process that is to be 
performed by the microcomputer. In this manner, when a function for 
automated evaluation is added, unstable factors in the evaluation increase and 
the quality of the application program AP to be evaluated lowers. 

20 Therefore, it is an object of the present invention to provide an automated 

evaluation system that can retain the quality of an application program and also 
maintain the stability in the automated evaluation. 

SUMMARY OF THE INVENTION 

25 An automated evaluation system in accordance with the present invention 

to solve the problems described above is provided, wherein the automated 
evaluation system ^febat-automatically evaluates an application program to be 
operated on a target system according to a given input event ar^d -bv comparing a 
reference output result corresponding to the given input event and a simulation 

30 resvilt generated by a simulator in response to the given input eventr . ie 
charactorizod in that it A n e valuator is accessible to a simulation apparatus that 
simulates an operation of the application program and comparoo a generates the 
simulation result corresponding to the given input events by tho oimtilation 
apparatvio — with Tthe reference output result is then compared to the 

35 corresponding simulation result to thereby perform an automated evaluation. 
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In accordance with the automated evaluation system that is accessible to 
the simulation apparatus, an automated evaluation can be performed using a 
simulation result corresponding to an input event for the simulation apparatuo . 
In this instance, the simulation apparatus performs a normal simulation of an 
5 operation of the application program based on the input event. Accordingly, 
there is no need to implement a function for automated evaluation in the 
application program. 

Furthermore, the automated evaluation system is equipped with a 
memory device that is accessible by both the simulation apparatus and the 
10 automated evaluation system, and characterized in that the cimulation 
apparatuo comparoe the simulation result stored in the memory device is 
compared w ith the reference output result. 

According to the automated evaluation oyotcm that io oqxiippod with a 
memory dovico that io accoooiblo by the automated evaluation syetom and the 
15 oimulation apparatuo, the automated evaluation oyBtom and the simulation 
apparatuo can be accoGOod by one another. The memory accessible bv both 
automated evaluation system and the simulation apparatus further enables the 
latter two components to communicate with one another. 

As a result, in an automated evaluation by the automated evaluation 
20 system and the simulation apparatus, a function for automated evaluation does 
not need to be implemented to perform the automated evaluation. 

Furthermore, the simulation apparatus and the automated evaluation 
system can be realized using the same computer. 

According to the automated evaluation system described above, the 
25 automated evaluation system and the simulator can communicate with each 
other regarding simulation results through a memory device provided in the 
computer, 

Aloo, a program in accordanco with tho T he present invention , in which 
feh ealso provides a program that r enders a computer to porform capable of 
30 performing an automated evaluation of a application program to be operated on 
a target system according to a given input event a^j4- -by comparing a reference 
output result corresponding to the given input event r with a simulation result 
generated in response to the given input event . 

The program is accessible to a simulation apparatus that simulates an 
35 operation of the application program, and 
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compares a simulation result corrosponding generated by the simulation 
apparatus in response to the given input event by the oimulatidn apparatuo with 
the corresponding reference output result to thereby render the computer able to 
perform an automated evaluation of the application program. 

5 In accordance with the program, which is made accessible to the 

simulation apparatus, an automated evaluation can be performed using a 
simulation result for an input event of the simulation apparatus. In this 
instance, the simulation apparatus performs a normal simulation of an operation 
of the application program based on the input event. Accordingly, there is no 
10 need to implement a function for automated evaluation in the application 
program, and the automated evaluation system can be realized by the computer 
while retaining the quality of the application program. 

Also, the program in accordance with the present invention is 
characterized in that it accooooo is accessible to t he simulation apparatus 
15 through an application programming interface implemented in an operating 
system of the computer. 

According to the program described above, the automated evaluation 

system and the simulation apparatus can be accoGood by oee 

anothor communicate using a function implemented in the operating system of 
20 the computer that executes the program. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a -an exemplary structure of a personal computer in which an 
automated evaluation system in accordance with ono ombodimont of the pro go nt 
25 invention and a simulation apparatus are operated in accordance with an 
embodiment of the invention . 

Fig. 2 is an illustration showing acccoooG access and communication 
between an automated evaluation system^ in accordance with the ombodimont of 
the proGont invention and a simulator , and a memory (RAM) in accordance with 
30 an embodiment of the present invention . 

Fig. 3 shows an example of input events and reference output results, 
wherein Fig. 3 (a) shows key inputs that arc (keystroke) input events, Fig. 3 (b) 
shows a reference output picture on an LCD before tho koy inputs shown in Fig. 
g-fa ^anv input event is executed, and Fig. 3 (c) shows a roforoncc rcoult, which is 
35 a reference output picture on the LCD for the key inputs shown in Fig. 3 (a). 
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Fig. 4 shows a conventional structure for performing an automated 
evaluation of an application program of a microcomputer. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

5 An automated evaluation system in accordance with an embodiment of the 

present invention is described below with reference to the accompanying 
drawings. ¥i^, — 1 ohowo a otructuro of a pcroonal computer in which an 
automated evaluation oyotom and a oimulation apparatuo arc oporatod. Fig. 2 is 
an illuotration to dcocribo accoGOGO botwcon the automated cvalviation oyotom 
10 and a oimulator. — Fig. 3 ohowo an oxamplo of input ovonto and roforonco output 
roGultG, whoroin Fig. 3 (a) ohowo key inputo, Fig. 3 (b) ohowG a roforoncc output 
picturo on an LCD before the key inputo ohown in Fig. 3 (a), and Fig. 3 (c) ohowo 
a roforonco output picture on tho LCD for the key inputo ohown in Fig. 3 (a). 

The automated evaluation system in accordance with the present 
15 invention automatically evaluates an application program to be operated on a 
target system uoing — a — oimulation — rooult — by — fefee — oimulation — apparatuob y 
comparing simulation results with respective reference results for respective 
input events . In this inotanco doing so, 47fe:e-a simulation apparatus performs a 
normal process, in which a simulation for an input event is performed and a 
20 result of the simulation is output. Therefore, a function for an automated 
evaluation does not need to be implemented in the application program. 

It is noted that the target system is operated based on an application 
program. The target system includes, for example, a microcomputer. Also, input 
events to be applied are set according to the specification of an apparatus in 

25 which the target system is implemented, and input events to be applied vary 
depending on input devices of an apparatus in which the target system i& 
implomontod apparatus . Input events to be applied are, for example, key inputs 
(e.g.. keystrokes), voice inputs, and the like. Reference output results are normal 
outputs of the target system according in response to input events, and are set 

30 for the specification of an apparatus in which the target system is implemented. 
Reference output results to be used vary depending on output devices of an 
apparatus in which the target system is implemented. Reference output results 
to be used include, for example, picture outputs, voice outputs, and the like. 

In accordance with thre — an embodiment of the present invention, the 
35 automated evaluation system is composed as an automated evaluation system 
that performs an automated evaluation through by loading an automated 
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evaluation program from a storage media onto a personal computer through a 
otoragG media that otoroo the automated evaluation pi^ogram and rendering the 
personal computer to oxocuto capable of executing an operation b ^in accordance 
with the avitomatic automated evaluation program. Furthermore, the automated 
5 evaluation system uooo accesses a disk apparatus that io connected toi n 
communication with the personal computer in order to take in t o obtain input 
events and reference output results. Also, in accordance with the present 
embodiment, the simulation apparatus is composed as a simulator that performs 
a simulation through b y loading a simulation program from a storage media onto 

10 the personal computer through a otorago media that otoroo the simulation 
program and rendering the personal computer to oxocuto capable of execviting an 
operation by the simulation program. It is noted that the automated evaluation 
system and the simulator may be composed within the same personal computer. 
Also, in accordance with the present embodiment, the target system is a 

15 microcomputer. Furthermore, in accordance with the present embodiment, the 
microcomputer can be implemented in an apparatus that can execute key 
(button) inputs from the exterior and picture outputs on an LCD, such as, for 
example, a gaming machine, a watch, a data bank and the like^ 

First, referring to Fig. 1, the entire structure of an automated evaluation 
20 system 1 (e.g., a validator) and a simulator 2 (e.g., an emulator) is described. 

In the automated evaluation system 1, an automated evaluation program 
is read from a storage media that stores the automated evaluation program fey-to 
a disk apparatus DU of a personal computer PC, where the automated 
evaluation program is loaded onto a RAM (Random Access Memory) (not shown)7 

25 ). and tho The automated evaluation program is then executed by a central 
processing apparatus CP to perform an automated evaluation. The disk 
apparatus DU may be an apparatus that can read to and write fe^ -from a storage 
media such as a floppy disk, an optical disk and the like. Also, the automated 
evaluation system 1 reads an input event stored in an input event file IF from in 

30 the disk apparatus DU, and transfers the input event to the simulator 2. 
Furthermore, the automated evaluation system 1 reads in a reference output 
result that is stored in a reference output file OF from in the disk apparatus DU 
and compares the same with Or -input-event-based simulation result for an input 
event o f generated by the simulator 2 to perform an automated evaluation. It is 

35 noted that the automated evaluation system 1 may store an evaluation result in 
the disk apparatus DU as a result log file, and outputs a picture on a display DP. 
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In the simulator 2, a simulation program is read from a storage media 
that stores the simulation program fey^to_the disk apparatus DU of the personal 
computer PC, where the simulation program is loaded on the RAM (not shown)7 
). and the The simulation program is then executed by the central processing 
5 apparatus CP to perform a simulation. Also, the simulator 2 reads an 
application program AP of the microcomputer from the disk apparatus DU, and 
simulates an operation by the application program AP. 

It is noted that, when an input event is transmitted by the automated 
evaluation system 1, the simulator 2 performs a simulation of the operation by 
10 the application program AP based on the input event. Then, the simulator 2 
stores a result of the simulation in a RAM 10 of the personal computer PC (see 
Fig. 2). 

Also, the simulator 2 can be externally operated by a keyboard KB 
corresponding to key inputs of the implemented apparatus, and outputs an LCD 
15 picture on the display DP corresponding to an LCD picture output of the 
implemented apparatus. 

The simulator 2 is connoctod to i n communication with a debugger 3, such 
that, while a simulation is performed, the application program AP can be 
debugged. With the debugger 3, a debug program is read from a storage media 

20 that stores the debug program b ^to t he desk apparatus DU of the personal 
computer PC, where t he program is loaded onto a RAM (not shown^). onad 
theThe program is then executed by the central processing apparatus CB to 
perform a debugging. The debugger 3 can start and stop the simulator 2, refers 
to data or rewrite data on the simulator 2. Furthermore, the debugger 3 can 

25 execute the application program AP at each step or break it. 

Next, the input event file IF and the reference output file OF are described 
below. The input event file IF and the reference output file OF can be created 
using an input event data creating function or a reference data creating function 
of the automated evaluation system 1, or may be created in advance by an editor. 

30 First, the case in which the input event file IF is created by the automated 

evaluation system 1 is described. First, keys of the apparatus in which the 
microcomputer is implemented are assigned to keys in the keyboard KB by the 
automated evaluation system 1. Then, the user prepares multiple input events 
according to the specification of the implemented apparatus and inputs them one 

35 by one using the keys. Then, the automated evaluation system 1 confirms the 
key type for each of the input events and the input order as input event data. 
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Finally, the automated evaluation system 1 stores the event data for the entire 
input events in the input event file IF. It is noted that the input event file IF is 
stored in a storage media with an optional filename being added and set in a 
state such that it can be read from the disk apparatus DU. Incidentally, the 
5 input event file IF can be modified according to changes in the specification of 
the microcomputer, changes in the specification of the implemented apparatus, 
changes in the evaluation contents^, and or the like. For example, as shown in 
Fig. 3 (a), let us assume that a key operation is conducted for one input event 20, 
in which a key [A] is depressed, a key [B] is depressed and a key [C] is depressed 
10 in this order. In this case, the input event 20 is stored in the input event file IF 
as input event data defining key types of the keys A, B and C and the key input 
order. 

Next, the case in which the reference output file OF is created by the 
automated evaluation system 1 is described. Since the reference output file OF 

15 stores reference data in which each one reference output result corresponds to 
each one input event, it is created corresponding to the creation of the input 
event file IF. Each time the user inputs an input event by operating a key, the 
automated evaluation system 1 transfers the key input to the simulator 2 as an 
input event. Then, the simulator 2 performs a simulation for the key input, and 

20 displays a simulation result on the display DP. Upon display, the user confirms 
the displayed content on the display DP, and confirms it as a reference output 
result if the content is correct. In this case, since an output device of the 
implemented apparatus is an LCD, the reference output result (reference data) 
includes image data to be displayed on an LCD and position data for displaying 

25 on the LCD. It is noted that when an application program AP is under 
development, a reference output file OF for upgrading the version of the 
application program AP may be created. In this instance, the reference output 
file OF can add corrections of bugs or changes in the specification and the like to 
the automated evaluation items of the application program AP so that the 

30 automated evaluation including the changes in the application program AP can 
be performed. Alternatively, after the user inputs an input event by a key 
operation, image data for displaying on the LCD may be formed as a reference 
output result by a reference^-data'-creating editor of the automated evaluation 
system 1. 

35 Finally, the automated evaluation system 1 stores reference data for the 

entire reference output results in the reference output file OF. It is noted that 
the reference output file OF is stored in a storage media with an optional 
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filename being added, and is set in the state such that it can be read from the 
disk apparatus DU. The file name of the reference output file OF is described in 
the input event file IF, and the reference output file OF is read according to the 
input event file IF. Therefore, the reference output file OF is modified according 
5 to the input event file IF. For example, as shown in Fig. 3 (b), let us assume 
that, before the input event 20 is inputted, a reference output picture 21 displays 
[ ] on its left upper end on the LCD. 

Then, when the keys are operated for the input event 20 as shown in Fig. 

3 (a), a reference output picture 22 displays [ABC ] starting from the left upper 

10 end to the right on the LCD as a reference output result 23. In this case, the 
reference output result 23 is stored as reference data including image data for 

displaying [ABC ] on the LCD and position data for displaying the image on the 

LCD in the reference output file OF. 

Next, referring to Fig. 2, operations of the automated evaluation system 1 
15 and the simulator 2 that take place when an automatic evaluation is performed 
are described. 

The automated evaluation system 1, when started by the user, loads input 
event data ID stored in the input event file IF from the disk apparatus DU onto 
the personal computer PC. It is noted that the input event file IF is designated 
20 by its file name by the user. When the input event file IF is loaded, the 
automated evaluation system 1 loads reference data RD stored in the reference 
output file OF having the filename described in the input event file IF onto the 
personal computer PC. 

Then, the automated evaluation system 1 transfers one input event from 
25 the input event data ID to the simulator 2. For transmission of the input event, 
an API [Application Programming Interface] command of the OS [Operating 
System] on the personal computer is used. For example, when the OS is 
Windows, one API command, i.e., FindWindow is used to obtain a window handle 
of the simulator 2. Then, another API command, PostMessage is used to 
30 transmit the one input event among the input event data ID to the window 
handle. In other words, the automated evaluation system 1 and simulator 2 can 
tranomit communicate with each other with respect to input events to each other 
by using API commands. It is noted that the automatic evaluation system 1 and 
the simulator 2 use the functions that are provided in the operating system, such 
35 as, API commands. Therefore, a special function for transmitting input events 
does not need be added. 
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Each time an input event is transmitted, the simulator 2 simulates an 
operation to be performed by the application program AP based on the input 
event. Then, the simulator 2 temporarily stores LCD display image data and 
position data as a simulation result in the RAM 10 for displaying on the display 
5 DP. Also, the simulator 2 displays the LCD display image data stored in the 
RAM 10 on the display DP. 

The process performed by the simulator 2 is the same as the normal 
process in which an operation by the application program AP is simulated, and a 
special process is not performed for the automated evaluation. Therefore, the 

10 application program AP that is the same as an application program actually 
mounted on a microcomputer can be used. It is noted that the RAM 10 is a built- 
in RAM in the personal computer PC, and is a RAM that is commonly shared by 
the automated evaluation system 1 and the simulator 2. Therefore, the RAM 10 
can be accessed from b y the automated system 1 and the simulator 2. In other 

15 words, the automated evaluation system 1 and the simulator 2 can communicate 
with each other with respect to simulation results through the RAM 10. Since 
the automated evaluation system 1 and the simulator 2 both use the RAM 10 of 
the personal computer PC, no special function needs to be added for referring to 
the simulation result. It is noted that the RAM 10 may be a VRAM of the 

20 personal computer. 

It is noted that, in the present embodiment, the RAM 10 
corresponds to a storage device. 

After the simulation, the automated evaluation system 1 refers to the 
simulation result stored in the RAM 10. Then, the automated evaluation system 

25 1 compares the simulation result and the reference data (RD). that is^ the 
reference output result (the LCD display image data and the position data) 
corresponding to the loaded input event in roforonco data RD. which io that was 
transmitted to the simulator 2. The automated evaluation system 1 makes a 
determination according to whether or not both of the results occur concur with 

30 each other, to thereby evaluate the operation of the application program with 
respect to the input event. Furthermore, the automated evaluation system 1 
stores the determination result in a result log file. The result log file may store 
all of the determination results, or may store determination results only when 
the simulation results differ from the reference output results. It is noted that 

35 the automated evaluation system 1 may display the simulation results and 
reference output results side -by-side on the display DP, so that the user can 
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confirm them. Also, the automated evaluation system 1 may display 
determination results on the display DP. 

Each time an evaluation for one input event is completed, the automated 
evaluation system 1 repeats the process described above for the next input event 
5 stored in the input event data ID to thereby carry out the automated evaluation. 
When the evaluation of all of the input events in the input event data ID is 
completed, the automated evaluation system 1 stores the evaluation result s in a 
log long file in a storage media such as a hard disk or the like according to the 
instruction by the user, and completes the automated evaluation. 

10 According to the automated evaluation system 1, the automated 

evaluation system 1 and the simulator 2 can fee — accGesod — fee — each 
etbej ^commimicate through API commands and the RAM 10 , which is accessible 
to both the system 1 and the simulator 2 . Therefore, for transferring input 
events from the automated evaluation system 1 to the simulator 2, or for 

15 referring by the automated evaluation system 1 to simulation results of the 
simulator 2, special functions do not need to be added to the automated 
evaluation system 1 or the simulator 2. Furthermore, the application program 
AP does not need to have a function for automated evaluation implemented 
therein, and can be the same as an application program that is actually mounted 

20 on the microcomputer. 

The present invention is not limited to the embodiments described above^ 
aad -rather, many modifications can be made within the scope of the appended 
claims . 

For example, API commands and the RAM 10 are used such that the 
25 automated evaluate system and the simulation apparatus can bo accoGsod 
^ ecommunicate with each other. However, other devices /functions can be used 
without — boing — limited — te — the — ombodimont — dovicoo to facilitate such 
communication . 

Also, the automated evaluation system and the simulator are composed on 
30 the same personal computer. However, they can be composed on other electronic 
processors such as workstations. 

Since the automated evaluation system in accordance with the present 
invention described above is capable of accoooincx to communicating with the 
simulation apparatus, an automated evaluation can be performed using 
35 simulation results for input events of the simulation apparatus. In this instance, 
the simulation apparatus only performs a normal simulation for the input 
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events. Accordingly, there is no need to implement a function for automated 
evaluation in the application program. Accordingly, the quality of the 
application program is equivalent to that of an application program mounted on 
a target system. 

5 Also, the automated evaluation system in accordance with the present 

invention is equipped with a storage device that can be accessed by the 
automated evaluation system and the simulation apparatus, such that the 
automated evaluation system and the simulation apparatus can bo accGOGod to 
each — etheji H^ommunicate . Therefore, in an automated evaluation by the 
10 automated evaluation system and the simulation apparatus, a function for 
automated evaluation does not need to be implemented. As a result, the stability 
in the automated evaluation can be maintained. 
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WHAT IS CLAIMED IS: 

1-5 (Cancelled) 

6. (Currently Amended) A system for evaluating an application program, 
comprising: 

5 an evaluator that reads event data for each input event and reference 

data for each corresponding otorod reference output result, and transmits the 
event data for each input event in response to a -an operating system command; 
and 

a simulator that receives the event data for each road input event from 
10 the evaluator, simulates an operation of the application program based on the 
event data for each input event, and outputs a corresponding simulation result 
corroGPonding to that in the form of simulation data for each input event; 

wherein the simulation data for each simulation result is compared with 
the reference data for the corresponding reference output result and a 
15 determination is made as to whether each simulation result is substantially the 
same as the corresponding reference output result, to thereby perform an 
automated evaluation of the application program with respect to the 
corresponding input event . 

7. (Previously Presented) The system of claim 6, further comprising a memory 
20 that is accessible by both the evaluator and the simulator, wherein the simulator 

outputs each simulation result to the memory. 

8. (Currently Amended) The system of claim 6, wherein the evaluator and the 
simulator are embodied in a single computer. 

9. (Previously Presented) The system of claim 8, wherein the evaluator is a 
25 program of instructions executable by the computer. 

10. (Currently Amended) The system of claim 9, wherein each operating svstem 
command is an application programming interface command implemented in an 
operating system of the single c omputer and through which the evaluator and 
the simulator communicate. 
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11. (Currently Amended) The system of claim 9, wherein each operating system 
command is a function implemented in an operating system of the single 
computer and through which the e valuator and the simulator communicate. 

12. (Currently Amended) The system of claim 6, further comprising an input 
5 event file that contains the event data for each input event, and a reference 

output file that contains reference data for each corresponding reference output 
result. 

13. (Currently Amended) The system of claim 12, wherein the evaluatdr stores 
the event data for each input event contained in the input event file. 

10 14. (Currently Amended) The system of claim 12, wherein the evaluator stores 
the reference data for each reference output result contained in the reference 
output file. 

15. (Previously Presented) The system of claim 6, further comprising a display, 
wherein the simulation result and the reference output result are displayed on 

15 the display. 

16. (Currently Amended) A method for evaluating an application program, 
comprising the steps of: 

(a) reading event data for each input event and reference data for each 
corresponding stored reference output result; 
20 (b) transmitting each the read event data for each input event in 

response to an application programming interface command; 

(c) simulating an operation of the application program based on the e vent 
data for each input event, and outputting a corresponding simulation result 
corrooponding to that in the form of simulation data for each input event; 
25 (d) comparing the simulation data for each simulation result with the 

reference data for the corresponding reference output result; and 

(e) determining whether each simulation result is substantially the same 
as the corresponding reference output result to thereby perform p erforming an 
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automated evaluation of the application program based on the comparison 
results in step (d). 
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